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ABSTRACT 



The purpose of this thesis is to analyze the feasibility 
of accrediting the Janus combat simulation for use with an 
Army concept called Model-Test-Model. Aircraft detection 
ranges from an operational field test are compared to similar 
ranges generated by the Janus combat simulation. Means and 
distributions of detection ranges for both helicopter and 
fixed-wing aircraft are studied. Methods of collecting 
detection data in field tests and representing aircraft in 
Janus must be improved for Janus to be accredited for aircraft 
detection range analysis of field tests. 
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I . INTRODUCTION 



A. THESIS OBJECTIVE 

The objective of this thesis is to validate or accredit 
the Janus combat simulation system for ground to air 
detections within the post-test phase of the Model-Test-Model 
(MTM) concept. The approach will be to compare field test 
detection range data of the Line of Sight-Forward-Heavy 
surface-to-air missile system with that of the Janus combat 
simulation . 

B. DESCRIPTION OF THE LINE OF SIGHT-FORWARD-HEAVY (LOS-F-H) 

SYSTEM 

For years, the Army has recognized the need for a capable 
low-altitude, short-range air defense weapon system to replace 
the outdated Chapparal and Vulcan systems. The LOS-F-H should 
eventually fill that need. 

The LOS-F-H is a surface-to-air system mounted on a 
modified Bradley fighting vehicle. The only major armament of 
the LOS-F-H is its missiles. The LOS-F-H carries eight 
missiles onboard. It is manned by a crew of three: a 
commander/radar operator (RO) , a gunner/electro-optics (EO) 
operator, and a driver. The LOS-F-H has its own acquisition 
and tracking radar which can track on several aircraft while 
scanning for others out to a range of twenty kilometers. The 
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crew uses a Forward Looking Infrared (FLIR) sensor or an 
optical (TV) sensor to track and help identify the target. 
Identification of the target is finalized using the 
Identification, Friend, or Foe (IFF) device, which receives an 
encrypted transmission sent out by friendly aircraft [Ref. 1] . 

The LOS-F-H missile is designated as a 'laser beam rider' 
because it is guided by a coded laser beam from the fire unit. 
The laser beam rider capability lessens the effectiveness of 
the enemy's use of countermeasures to disrupt missile flight 
through radar jamming. 

The LOS-F-H weapon system weighs approximately 31 tons 
fully loaded, including eight 112 pound missiles in ready-to- 
fire canisters. Figure 1 is a picture of the LOS-F-H and the 
crew positions. 
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C. DEFINITION OF DETECTION 

For the Janus trials, aircraft detection occurs when the 
LOS-F-H radar determines that some type of airframe is 
approaching, records its location, and identifies the target 
as friendly or hostile. The field test trials require a 
further step for aircraft detection: a crewman must be aware 
of the approaching target. The differences in these two 
processes make their comparison difficult. Though hostile and 
friendly aircraft detections were recorded in both the field 
test and Janus, our concern will be with detections of hostile 
aircraft only. 

D. MODEL-TEST-MODEL CONCEPT 

Field tests are fundamental to the acquisition of any new 
weapon system. Critical to the quality of a field test is 
that it accurately portrays the capabilities of the system 
within its operational context. One method that will help 
improve the quality of field tests is Model-Test-Model. 

The term Model-Test-Model refers to the three primary 
phases: the pre-test phase, the field test, and the post-test 
phase . 

1 . PRE-TEST PHASE 

The pre-test phase involves use of a combat simulation 
model by members of the field test evaluation team, the Army 
proponent, the modellers, and maneuver unit leaders. During 
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this phase, scenarios are developed within the simulation that 
can be used during the field test. The scenarios must 
incorporate the restrictions placed on the test, including 
representation of the terrain, numbers of players, and the 
force mix. The maneuver unit leaders try to determine tactics 
for each scenario, such as routes of march and defensive 
positions. The results of the scenario development are used 
by the field test evaluation team in developing and improving 
their data collection and analysis methods and to improve the 
conduct of the field test. Tactics that were successful in 
both the model and field test should be provided to the Army 
proponent to update doctrine. 

The pre-test phase also can provide training for the 
maneuver unit leaders. Many past tests have indicated that 
several trials in a field test may be necessary for player 
units to get adequately organized and trained. Exposure to 
the scenario, terrain, and systems that will be involved in 
the field test during the pre-test phase may make the earlier 
field trials better, since the maneuver leaders are presumably 
familiar with major facets of the field test from the outset. 

2. FIELD TEST PHASE 

During the field test phase, the system is tested under 
realistic combat conditions to determine its operational 
effectiveness and suitability. It is important that typical 
users of this system are used in the field test and that the 
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threat forces employed are realistic. The specified minimum 
acceptable operational requirements delineated by the system's 
proponent must be adequately tested. 

The use of MTM in the pre-test phase allows the field 
testers to use improved scenarios derived from pre-test 
modelling. The amount of replication of the pre-test tactics 
employed may differ for force development tests and 
operational tests. 

Force development tests are conducted early in the weapon 
acquisition process and are meant to examine the effectiveness 
of existing or proposed concepts of training, doctrine, 
organization, and material. Therefore, force development 
tests should include trials which attempt to replicate as 
closely as possible some of the tactics developed by the 
maneuver unit leaders during the pre-test modelling. This 
provides field test results for some of the tactics examined 
in the simulation and helps determine if these successes can 
be approximated in the testing environment. 

Operational tests, both initial and follow-on, occur later 
in the acquisition process and are less controlled than force 
development tests. They focus on the operational 
effectiveness of new systems and whether they are worthy of 
production. Operational tests differ from force development 
tests in that tactical realism is critical and ground 
commanders do not follow a 'script'. Rather, maneuver leaders 
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use their own judgements in an operational test, with the aid 
of the knowledge gained from the pre-test phase. 

3 . POST-TEST PHASE 

Following the field test, the post-test phase begins. The 
immediate goal of the post-test phase is to validate or 
accredit the simulation model. Since we are primarily 
concerned with post-test model accreditation, the process for 
accomplishing this goal is detailed in this thesis. 

As a second goal of this phase, evaluators can use the 
improved simulation for cautious predictions. Constraints of 
the field test, such as numbers of players, dimensions of the 
maneuver areas, and types of terrain, could be altered within 
the simulation. Results of runs with the modified simulation 
model could add valuable information for consideration by the 
evaluator. 

MTM combines force-on-force battle simulation and field 
testing with the goal of improving them both. MTM is used to 
support test design and evaluation of test results. 

4. DEFINITION OF VALIDATION AND ACCREDITATION 

Before a model can be used to represent an actual field 
test, it must be shown that the model is a measureably close 
representation of that field test. The degree to which it can 
be shown to represent the field test will determine whether 
the model is validated or accredited [Ref. 2]. 

The concepts of model validation and accreditation were 
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addressed by Walter Hollis, the Deputy Undersecretary of 
Defense for Operations Research. He defines model validation 
as follows. 

The process of determining that a model is an accurate 
representation of the intended real-world entity from the 
perspective of the intended use of the model [Ref. 3]. 

In this thesis, our intent is not to attempt to show that the 

model is an accurate representation of the LOS-F-H field test, 

as we are concerned only with analyses of aircraft detection 

ranges . 

On the other hand, accreditation is concerned with more 
specific applications. Mr. Hollis defines accreditation as 
follows . 

Certification that a model is acceptable for use for a 
specific type of application. Accreditation is possible 
even if the model is not fully validated [Ref. 3]. 

Determining that a model accurately represents all aspects 
of a field test may be difficult. Accreditation, or 
demonstrating that a model is accceptable for certain 
applications, such as a comparison of detection ranges, might 
be feasible. 

E. HOW THIS THESIS FITS INTO MODEL-TEST-MODEL 

In this thesis, we are concerned strictly with 
accreditation of the Janus model. Specifically, we compare 
aircraft detections by the Line of Sight-Forward-Heavy (LOS-F- 
H) from an actual field test conducted in May and June of 1990 
with simulations of the selected test trials with the Janus 
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model . 



Only detections by LOS-F-H systems of fixed wing 



aircraft and helicopters are analyzed. In the 
chapters we discuss the LOS-F-H field test and 
simulation model. 



next two 
the Janus 
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II. CONDUCT AND DESCRIPTION OF THE FIELD TEST 



A. DESCRIPTION OF THE FIELD TEST 

1. FIELD TEST BACKGROUND 

The purpose of the field test was to evaluate the 
operational effectiveness and suitability of the LOS-F-H 
platoon in performing its designated role of providing air 
defense for heavy maneuver forces in the forward area. 

The agency conducting the field test was TRADOC Test and 
Experimentation Command, Experimentation Center (TEC) . The 
test was an Initial Operational Test and Evaluation (lOTE) . 
For the sake of simplicity, we will refer to it as the 'field 
test' throughout this document. 

Over 1000 personnel participated in the field test, and 
almost 600 of them were members of tactical units from the 
Army, Air Force, and the Air National Guard. The field test 
took place at Fort Hunter Liggett, California in the spring 
of 1990. 

2. DESCRIPTION OF INSTRUMENTATION 

All ground vehicles and aircraft had sophisticated 
instrumentation packages installed. This equipment consisted 
of laser sensors and emitters which were intended to provide 
updates of battle progress. This capability is called real- 
time casualty assessment (RTCA) . With this instrumentation on 
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each vehicle, player locations, as well as important events 
such as engagements, could be monitored and recorded. Much 
data gathering and analysis depended on the accuracy of this 
process . 

3. FIELD TEST PHASES 

The field test consisted of three phases: the pretest, 
exploratory trials, and record trials. The pretest began in 
late February 1990 and lasted until mid March 1990. During 
this time, personnel involved in the field test finalized 
individual training on their weapons. Additionally, all RTCA 
instrumentation packages were installed on the vehicles. 

A total of 15 exploratory trials were conducted over an 
eight day period [Ref. 1] . The objectives of the exploratory 
trials included allowing maturation of the data collection and 
reduction procedures, resolving instrumentation problems, and 
providing an opportunity for test controllers and players to 
refine their procedures. Additionally, the exploratory trials 
provided objective evidence about whether instrumentation and 
players were ready to start record trials. 

Fifty record trials were conducted from 9 April to 23 May 
1990. The trials were conducted in the same manuever area, 
which had dimensions of approximately 15 kilometers by 15 
kilometers. Each trial was a force-on-force battle which 
generally lasted about one hour. Normally, two trials were 
conducted each test day. The friendly units were designated 
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as Blue forces and the 'threat' units as Red forces. The 
battles involved Red and Blue mechanized forces of battalion 
strength. Table I shows the typical size and make-up of Blue 
and Red forces during a battle. For Red weapon systems, 
American made 'surrogates' were used. The surrogates are 
identified in the table in parentheses. 



Table I BLUE AND RED FORCES IN LOS-F-H FIELD TEST 



BLUE PLAYER 


NUMBER OF SYSTEMS 


MlAl TANK 


10 


M3 CAVALRY FIGHTING VEHICLE 


12 


AH-1 COBRA HELICOPTER 


2 


OH-58 KIOWA HELICOPTER 


3 


A- 10 THUNDERBOLT ATTACK PLANE 


2 


LINE-OF-SIGHT FORWARD HEAVY 


4 



RED PLAYER (SURROGATE) 


NUMBER OF SYSTEMS 


FUTURE SOVIET TANK (M60A3 TANK) 


10 


SOVIET BMP (M113 ARMORED 
PERSONNEL CARRIER) 


12 


HIND HELICOPTER (AH-64 APACHE) 


3 


HAVOC HELICOPTER (AH-64 APACHE) 


1 


HIP HELICOPTER (UH-60 


4 


BLACKHAWK) 




FROG FOOT ATTACK PLANE (A- 7 


2 


CORSAIR) 
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The LOS-F-H mission during all record trials was to defend 
the Blue maneuver force against air attack as the force 
conducted its mission. The Red forces had no comparable Air 
Defense Artillery asset. 

4. FIELD TEST CONDITIONS AND THE SELECTION OF TRIALS 
The conditions of the trials were varied. These 
conditions included north or south orientation, day or night, 
defense or offense, and the Mission Oriented Protective 
Posture (MOPP) of either 0 or 4 . MOPP 4 indicates that all 
players must wear full protective chemical garments. 

We did not attempt to make analyses related to all 50 
field trials. Only four trials were analyzed, due to their 
identical conditions of daytime, defense, MOPP 0, and Blue in 
the South, and because these trials had the most helicopter 
and fixed-wing detection data available. 

B. FIELD TEST RESOURCE LIMITATIONS 

There were two resource limitations that could have 
impacted on the quality of the data resulting from the field 
test. One was the lack of artillery in any of the trials. 
The use of artillery might have caused the LOS-F-H systems to 
move more often, thereby reducing their effectiveness; 
artillery most likely would have destroyed several LOS-F-H 
systems. The result would have been fewer enemy aircraft 
detected. 
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A second limitation was a safety precaution that allowed 
fixed-wing pilots to fly no lower than 200 feet above the 
ground. The 'ground' was interpreted by the pilots to be the 
valley floor, and therefore meant they could fly closer to 
tree-top level when over hilly areas, since they would still 
be 2 00 feet above the valley floor. This precaution made 
detection of enemy fixed-wing aircraft more likely, since they 
were reguired to fly higher than what they probably would fly 
in combat. Higher flight altitude makes the aircraft more 
vulnerable to ground radar, as it has less chance of being 
masked by mountains or other terrain features. 
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III. DESCRIPTION AND USE OF JANUS SIMULATION 



A. OVERVIEW OF JANUS 

Janus is an interactive, force-on-force, high-resolution 
combat model used extensively throughout the Army. The 
original version of Janus was developed at the Conflict 
Simulation Center at Lawrence Livermore Laboratory for the 
purpose of creating a two-sided analytical and training tool 
to study the dynamic battlefield. It was later improved by 
the Janus Working Group at Tradoc Analysis Command (TRAC) , 
White Sands Missile Range. Janus is intended for use at 
brigade-level and below. The Janus code is written in FORTRAN 
[Ref. 4]. 

The Janus simulation models combat systems, such as tanks 
and helicopters, the battlefield environment, and each 
system's interaction with other systems and their 
environments. The characteristics of these combat systems 
include descriptions of the weapons carried, weapon 
capabilities, and much more. 

B. DEVELOPMENT OF THE JANUS TRIALS 

1. LOADING THE START POSITIONS AND ROUTES 

Prior to simulating a field test trial in Janus, it was 
necessary to enter the start locations and movement routes of 
all players. These start locations and movement routes were 
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the only data translated directly from the field test to the 
Janus trials. This was done using a combination of FORTRAN 
programs, written at TRAC-Monterey and TRAC -White Sands by Mr. 
A1 Kelner and CRT Allen East that rapidly loaded the start 
locations and routes for each trial. 

2. DESCRIPTION OF JANUS UTILITIES 

Before addressing the specific development of the trials, 
we describe how Janus can be manipulated by the user. This is 
accomplished using four user-friendly utilities built into the 
simulation software. The four utilities are the symbol 
editor, the PH/PK (probability hit/probability kill) editor, 
the graphic terrain editor, and the scenario editor [Ref. 4]. 

The symbol editor allows the creation or definition of the 
graphic symbols used by Janus to mark unit locations. Symbols 
may be drawn as simple standard military graphics or as very 
detailed silhouettes. The symbol editor contains many pre- 
defined symbols that the user may select. 

The PH/PK editor allows editing of data used to determine 
lethality over range for shooter-target pairs. Not only is 
range varied in these data sets, but also the postures of the 
shooter and target. For example, PH/PK varies for a moving or 
stationary shooter, as well as with a moving or stationary 
target. A specific Janus file can contain up to 1000 PH/PK 
data sets; each set represents the hit and kill probabilities 
of a particular shooter against a particular target. 
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The graphic terrain editor allows input and modification 
of terrain features, such as roads and vegetation, on a three 
dimensional terrain grid. The terrain resolution used in the 
analysis was 50 meters. This 50 meter resolution means that at 
every 50 meter interval in both an east/west and north/south 
direction, a terrain height reading was made and elevations 
between adjacent data points were interpolated to provide a 
relief representation [Ref. 5]. Janus can be used with higher 
or lower terrain resolution as well. All Janus map displays 
use digitized data from the Defense Mapping Agency and land 
satellite data. 

The scenario editor provides access to system/weapon 
characteristics, battlefield descriptive parameters, and 
scenario force structures. Up to 198 different systems can be 
defined and their characteristics can be varied in numerous 
ways . 

3. DEVELOPMENT OF JANUS SCENARIOS AND TRIALS 

Only the terrain editor and scenario editor were used 
extensively in Janus scenario development. The Fort Hunter 
Liggett terrain representation we used for our Janus trials 
was drawn from the Janus database. The graphic terrain editor 
allowed two improvements in the terrain representation that 
were based on personal knowledge of the area. One 
improvement was the addition of roads, which allowed vehicles 
to travel along roads in the simulation as they did in the 
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field test. The other change was the addition of vegetation 
that was missed in the database, yet was critical to proper 
representation of the maneuver area. 

The scenario editor was used extensively to create 
realistic Janus scenarios. The most critical concerns were 
accurately constructing the parameters of the LOS-F-H system 
and recreating the flight behavior of the fixed-wing aircraft 
and helicopters. Important LOS-F-H parameters included radar 
and sensor type, maximum detection range, detection 
probability of the LOS-F-H at certain ranges against various 
aircraft, and maximum engagement range. Critical aircraft 
parameters were flight speed and flight altitude. The values 
of these parameters were derived from the field test reports, 
conversations with the field test coordinator and two Janus 
programmers, and, where necessary, personal judgement [Ref. 6], 

C. UNCERTAINTY OF PARAMETERS 

While most of the data described above were available, 
certain information was not. Most significant among missing 
data were the probabilities of detections for the LOS-F-H 
against different types of aircraft. These values were 
estimated using guidance from a White Sands Missile Range 
Janus programmer and personal judgement [Ref. 6]. 
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IV. ANALYSIS OF FIELD TEST AND JANUS DETECTION DATA 



A. REASONS FOR DIFFICULTY IN COMPARING DETECTION DATA 

Research has revealed that comparing detection data from 
the field test to detection data generated in Janus will be 
difficult. One reason is the difficulty in measuring 
detections in a field test. The Real Time Casualty Assessment 
(RTCA) system cannot measure aircraft detections by individual 
soldiers manning equipment. Additionally, shortcomings in 
Janus and the field test restrict the ability to duplicate 
the field test environment. These problems are discussed in 
detail below. 

1. FIELD TEST DETECTION DATA GATHERING PROCESS 
As described in Chapter II, the Real Time Casualty 
Assessment (RTCA) system was effectively used to gather much 
data during the field test. However, the RTCA system could 
not provide detection data. According to Major Larry Dubois, 
the test coordinator from TEC, LOS-F-H detection data was 
acquired solely through analysis of video camera records. 

Each LOS-F-H system had a video camera recording the 
crew's actions and words throughout the trial. The video 
camera was placed behind the crew and revealed their actions 
and verbal communications. Personnel doing the reconstruction 
selected 'target handoff as an overt action to indicate that 
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a target was about to be engaged. 'Target handoff occurred 
when the radar operator (RO) designates one of the targets 
under track to the gunner/electro-optics (EO) operator for 
engagement. This could be verified by a red button, visible 
on video, that illuminated when 'target handoff occurred. By 
observing the time into the trial set in the recording, the 
range of the aircraft at 'target handoff was determined by 
the video reviewer by obtaining the location of the LOS-F-H 
and the aircraft from the movement file. Detections were then 
determined by backtracking from 'target handoff to another 
overt action, generally a verbal cue, alerting the video 
reviewer that a detection was made at that time. The range of 
detection was then calculated similarly to 'target handoff 
range . 

This process has several problems. First, only those 
detections that led to engagement, as indicated by 'target 
handoff, were noticed. A significant amount of data may have 
been lost, since many detections likely occurred which were 
never recorded because the particular LOS-F-H involved may not 
have been able to engage each target it detected. 
Additionally, the recorded detection ranges will have random 
errors, since the ranges are not actual, but rather were 
estimated by the video reviewers who depended on an overt sign 
rather than an actual detection. Finally, recorded detection 
ranges will tend to be shorter than detection ranges actually 
were. As mentioned earlier, detection occurs when a LOS-F-H 
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system recognizes a target and identifies it as hostile. The 
time of this event cannot be determined through analysis of 
the test data. This overt sign of detection occurred at an 
unknown time after the LOS-F-H detected the target, and the 
approaching target moved an unknown distance during this time. 
This problem must be taken into consideration before any 
conclusions can be drawn from detection range analyses. 

Another limitation of the test range instrumentation is 
its inability to determine aircraft elevation. Knowing the 
altitudes of fixed-wing aircraft and helicopters during the 
field test is important for accurately representing the field 
test in a simulation. 

2. AIRCRAFT FLIGHT REPRESENTATION IN JANUS 

The most significant problem in attempting to replicate 
the field test in Janus had to do with representing aircraft 
flight characteristics. An actual attacking aircraft has the 
ability to quickly change altitude and speed in order to 
confuse enemy air defenses or to deliver its ordnance and 
quickly leave the area. A critical limitation of Janus is 
that each aircraft can have only two flight speeds and two 
flight altitudes which most likely cannot closely replicate 
the actual attack profile of an aircraft. 

Additionally, there are two basic flight modes that an 
actual aircraft may use. One is 'constant altitude', in which 
an aircraft remains generally at a set altitude above a fixed 
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level on the ground (often sea level) , regardless of changing 
terrain elevation. Fixed-wing aircraft in the field test used 
a version of this method, flying generally at a constant 200 
feet above the level of the valley floor. The other flight 
mode an aircraft may use is called 'nap of earth' or terrain 
following. The altitude of an aircraft using this mode 
follows the changing elevation of the terrain. During the 
field test, helicopters used this tactic. 

Only the 'nap of earth' mode of flight is available in 
Janus, and therefore maintaining level flight paths for fixed- 
wing aircraft had to be approximated. This was done by using 
both available altitude settings, one set at 200 feet (64 
meters) for any portion of a flight path segment over a valley 
floor and the other set at 50 feet (16 meters) for a flight 
path segment over mountains. The height of 50 feet was chosen 
since the average elevation of the mountainous terrain above 
the valley floor is 150 feet. 

This limitation of Janus would seem to have great impact 
on fixed-wing detection data. Fixed-wing aircraft in Janus 
cannot fly steadily at 200 feet altitude as the fixed-wing 
aircraft generally did in the field test. Additionally, the 
approximation described above relies on an inexact estimation 
of the average elevation over mountainous terrain. Yet the 
results of a sensitivity analysis indicates that the effects 
of this estimation may be insignificant. 
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A data sample of detection ranges containing three 
iterations of one trial with the average elevation over 
mountainous terrain set at 80 feet (26 meters) were compared 
to a data sample of detection ranges of three iterations of 
the same trial using the original elevation over mountainous 
terrain of 50 feet (16 meters). The method of comparison was 
a two-sided Mann-Whitney test. These two data samples were 
combined and the detection ranges were ranked. The average 
ranks of the two data samples were close at 57.16 and 63.23. 
Due to a sufficiently large sample size of over 100, the Mann- 
Whitney Z statistic is applicable. The null hypothesis is 
that the distributions of the two data sets are equal. The Z 
statistic for this test is .949, and at a significance level 
of .05, the null hypothesis cannot be rejected. The 
STATGRAPHICS 5.0 software package was used in this and all 
other analyses in this thesis [Ref. 7] 

Therefore, while the availability of only a 'nap of earth' 
flying mode in Janus makes a comparison of fixed-wing 
detection data flawed, it appears that the approximation used 
does not add significant additional error. 

B. T^ALYSIS OF FIXED-WING DATA 

The fixed-wing data used in our analysis came from four 
similar trials. Each trial was conducted during daylight 
hours, had the Blue forces in the defense, and was conducted 
in a chemical and nuclear free environment. 
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The field test data contained at most one detection per 
pass through the manuever area for every fixed-wing aircraft. 
The planes always flew in pairs and made four or five passes 
during each trial. Each trial, therefore, had eight to ten 
detections recorded. 

Each set of Janus data had some repeated detections. To 
account for possible dependence between ranges of repeated 
detections, these repeated data points were eliminated, so 
only the initial detection ranges remained. No Janus 
iteration had more than six repeated detections. 

A summary of the sample sizes of fixed-wing data is 
displayed in Table II below. The number in each box reflects 
the sample size in each category. The sample sizes of Janus 
trials are a result of five independent iterations per trial, 
while the field test data comes from only one iteration. 



Table II FIXED-WING DETECTION SUMMARY 





TRIAL A 


TRIAL B 


TRIAL C 


TRIAL D 


TOTAL 


JANUS* 


45 


78 


79 


54 


256 


FIELD 

TEST 


10 


9 


12 


10 


41 


TOTAL 


55 


87 


91 


64 


297 



* 5 Janus Runs Pooled 
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A comparison of the fixed-wing detection data is displayed 
in Figure 2. This series of box-plots allows comparisons of 
the Janus and field test data from each trial side-by-side. 
The boxes show the median of each data set as well as its 
inter-quartile range. This plot shows no strong evidence of 
a difference in means between Janus and field test fixed-wing 
data of a common trial. 
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Figure 2 Fixed-Wing Box Plots 
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A Two-Factor Analysis of Variance (ANOVA) was used to test 
for equivalence of means of detection ranges between trials 
and between Janus and field test data for fixed-wing aircraft. 
The ANOVA summary table is shown in Table III. The two 
factors, or sources of variation, are labeled as trials and 
data source, and are related to testing the difference between 
trials and the difference between multiple Janus iterations 
and the field test data. 



Table III ANOVA TABLE FOR FIXED-WING AIRCRAFT 



SOURCE OF 
VARIANCE 


SUM OF 
SQUARES 


DF 


MEAN SQUARE 


F-RATIO 


SIG 

LEVEL 


TRIALS 


97,571,767 


3 


32 , 523 , 922 


3 . 627 


. 0135 


DATA SOURCE 


275 


1 


275 


. 000 


. 9956 


INTERACTIONS 


40,502,645 


3 


13,500,882 


1.506 


.2132 


RESIDUAL 


2,596,280,000 


289 


8,967,573 






TOTAL 


2,829,300,000 


296 









STATGRAPHICS 5.0 has two methods of computing sums of 
squares for the ANOVA table. The method we used, designated 
as Type III, computes the additional sum of squares for each 
factor as if it were added to the model last. This is the 
method recommended for use with an unbalanced design [Ref. 7]. 

Between the four different trials, the F-ratio was 3.627 
(df=3, 289) with a significance level of .0135, indicating a 
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significant difference between means. Between Janus and field 
test data, a very small F-ratio resulted in a significance 
level of .9956, revealing a strong indication of no 
significantly different means. The interaction of the two 
effects is insignificant as seen by its F-ratio of 1.506 
(df=3, 289) and significance level of .2132. However, the 
plot seen in Figure 3 below suggests some possible 
interaction. This conflict between the F-test and 
interactions plot could be a result of the small sample sizes 
of the field test data. But the apparent crossover may be due 
to random error; the standard error of the means for field 
test data for each trial is approximately 1000 meters, while 
that for Janus data for each trial is approximately 375 
meters. The standard error was calculated by dividing the 
standard deviation of every field test or Janus detection data 
set by the number of detections in that particular data set. 
Overall, we conclude there is not a significant interaction 
between trials and source (test or Janus) . 
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To check the adequacy of the ANOVA model for fixed-wing 
aircraft data, the data were tested for normality in 
distribution of residuals and for equality of variance. The 
histogram plot of residuals versus a normal curve is shown in 
Figure 4. It indicates an apparent lack of normality of the 
residuals. A Chi-Square Goodness of Fit Test of the fixed- 
wing data gives a chi-square statistic of 58.47 (df=16) . The 
corresponding significance level was .000, verifying that the 
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residuals do not appear to be normally distributed. This is 
not a great concern, due to the robustness of the ANOVA with 
regard to normality [Ref. 8]. Nevertheless, the distribution 
of the residuals of the log of the fixed-wing data were 
examined and did not appear normal. Ratios of sample 
variances between corresponding Janus and field test trials 
and from trial to trial resulted in values close to one. 
Additionally, 95% confidence intervals for ratios of variance 
include the value of one for each of the trials. Therefore a 
hypothesis of equality of variance cannot be rejected. 




Figure 4 Histogram Of Fixed-Wing Residuals Versus Normal Plot 
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An additional Two-Factor ANOVA was conducted using the 
four trials versus the individual Janus runs and the field 
test. The purpose of this test was to examine the effect of 
the individual Janus runs and any resulting residuals. This 
was accomplished by separating the data source factor into its 
5 Janus and a field test level. Table IV displays the results 
of this ANOVA. 

Table IV ANOVA OF FIXED-WING DATA BY INDIVIDUAL JANUS 
ITERATION 



SOURCE OF 
VARIANCE 


SUM OF 
SQUARES 


DF 


MEAN SQUARE 


F-RATIO 


SIG 

LEVEL 


TRIALS 


169,800,000 


3 


56,600,498 


6.203 


. 0004 


DATA SOURCE* 


36,629,000 


5 


7,325,743 


. 803 


. 5485 


INTERACTIONS 


96,229,338 


15 


6,415,289 


• 703 


• 781 


RESIDUALS 


2 , 491,200,000 


273 


9,125,252 






TOTAL 


2,829,300,000 


296 









* 5 Individual Janus Iterations and the 
Field Test Looked at Separately 



The significance levels above verify the previous ANOVA, 
in that the difference in means between trials is quite 
significant, while the differences in means between Janus 
iterations and the field test may not be significant. 
Additionally, the ANOVA shows no interactions between the two 
sources of variance. 
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C. ANALYSIS OF HELICOPTER DETECTION DATA 

The helicopter detection data we considered are for the 
same four trials as above for the fixed-wing case. Each field 
test trial contained approximately 30 helicopter detections, 
while five iterations of each Janus trial resulted in about 
100 detections when pooled. Both sets of helicopter data 
contained repeated detections of a given helicopter, as in the 
Janus fixed-wing data, and to avoid dependency among detection 
ranges, all were removed but the first of multiple repeated 
detections. Table V below shows a summary of the counts of 
helicopter detection ranges. 

Table V SUMMARY OF HELICOPTER DETECTION RANGES 





TRIAL A 


TRIAL B 


TRIAL C 


TRIAL D 


TOTAL 


JANUS* 


95 


147 


99 


98 


439 


FIELD 


29 


40 


19 


22 


110 


TEST 












TOTAL 


124 


187 


118 


120 


549 



* 5 JANUS RUNS POOLED 



A graphical comparison of the helicopter data is indicated 
by the multiple box plots in Figure 5. The differences 
between Janus and field test ranges is obvious, as the Janus 
ranges are larger than the field test ranges in all cases. 
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COMPARISON OF HELICOPTER DATA 
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Figure 5 Comparison of Helicopter Detection Ranges 



The helicopter detection data were compared with a Two 
Factor ANOVA. The ANOVA summary table is shown in Table VI. 
The two factors are the data source (Janus or field test) and 
the trials. This ANOVA gave large F-ratio values for trials 
and data source, indicating that there are significant 
differences in the means related to both factors. The F-ratio 
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for interactions between trials and data sources is 3.493 
(df=3, 541) with a significance level of .0155, indicating 
there is significant interaction between trials and data 
sources. However, a plot of these interactions, seen in 
Figure 6, shows the interaction is due to only trial 4. 

Table VI ANOVA TABLE FOR HELICOPTER DATA 



SOURCE OF 
VARIATION 


SUM OF 
SQUARES 


DF 


MEAN SQUARE 


F-RATIO 


SIG 

LEVEL 


TRIAL 


132,660,000 


3 


44,222,000 


10.230 


.000 


, 

DATA SOURCE j 


442 , 580,000 


1 


442 , 580,000 


102 . 387 


. 000 


1 INTERACTIONS 


45,289,872 


3 


15,096,624 


3.493 


.0155 1 


RESIDUAL 


2 ,338 , 500,000 


541 


4 , 322,579 






TOTAL 


3,242,600,000 


548 









32 



HEL 1 COPTER 



.oJ nteract I 

.or ^ ^ 



q 






c 

cr 



o 

I — 
LJ 
LU 
»— 
LU 
C=> 

2: 

C 




DATA SOURCE 

Figure 6 Helicopter Interaction of Means. Number Signify 
Trial . 



The adequacy of the Two-Factor ANOVA was checked by 
testing whether the residuals have a normal distribution and 
whether the detection ranges have common variance. Figure 7 
shows a graphical comparison of residuals to a normal curve. 
It suggests the residuals are not normally distributed. A 
Chi-Square Goodness of Fit Test verified this lack of 
normality, as the chi-square statistic is 87.93 (df=19) and 
has a significance level of .000. However, the robustness of 
the ANOVA test regarding lack of normality in the residuals 
makes it still valid but not as powerful [Ref 8] . 
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Figure 7 Helicopter Residuals Versus Normal Plot 

The ratios of variances between each set of corresponding 
Janus and field test data indicate common variance per trial. 
Additionally, scatterplots of detection means versus standard 
deviations for each set of helicopter data, shown in Figure 8, 
indicate the similarity of standard deviations and the lack of 
any strong relationship between the mean and standard 
deviation. Therefore, the assumption of common variance 
appears valid. 
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Figure 8 Means Versus Standard Deviation For Helicopters 



An additional Two-Factor ANOVA was again run with the 
helicopter detection data using trials as one factor and data 
sources as the other factor. For this test, however, data 
source is split into the five levels corresponding to Janus 
iterations per trial; the sixth level of this factor is the 
field test. The resulting ANOVA summary is shown in Table 
VII. 
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Table VII ANOVA TABLE OF HELICOPTER DETECTION DATA 



SOURCE OF 


SUM OF 


DF 


MEAN 


F-RATIO 


SIG 


VARIATION 


SQUARES 




SQUARES 




LEVEL 


TRIALS 


358,750,000 


3 


119,580,000 


27.31 


.000 


DATA SOURCE* 


452 , 840,000 


5 


90,567,000 


20.68 


.000 


INTERACTION 


69,738,112 


15 


4,649,207 


1.06 


. 389 


RESIDUALS 


2,298,800,000 


525 


4,378,711 






TOTAL 


3,242,600,000 


548 









* 5 INDIVIDUAL JANUS ITERATIONS AND 
FIELD TEST EVALUATED SEPARATELY 



Trials and data source factors are again highly 
significant. However, the interaction between trials and data 
sources is insignificant in this case, in contrast with the 
model used in connection with Table VI. This suggests that 
when the variability among Janus runs is accounted for, the 
apparent interactions become insignificant. The increase in 
interaction sum of squares in Table VII is more than offset by 
the increase in degrees of freedom. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 



A. CONCLUSIONS 

The analyses of a limited set of data show that that Janus 
and field test aircraft detection ranges do not agree. Our 
conclusion is, therefore, that we are not able to validate or 
accredit Janus for use with Model-Test-Model at this time. 
This is due in part to the difficulty in replicating the field 
test process in Janus. 

The comparison of fixed-wing data showed differences in 
the means of detection ranges between trials, but no 
significant difference between sources. Yet, the meaning of 
this result is not clear, since both sets of data being 
compared were limited. Additionally, the powers of the tests 
are low due to high variance and low field test sample sizes. 

B. RECOMMENDATIONS 

We have found that an attempt to replicate a field test 
trial in a simulation such as Janus requires a thorough 
knowledge of the field test by the modeler. Detailed 
information such as flight altitudes, speeds, movement routes 
used by aircraft and vehicles on the ground, and other 
characteristics not currently provided in field test data are 
needed. To ensure this is done, modelers and testers should 
closely coordinate prior to and throughout the field test. 
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Additionally, setting correct parameters within Janus is 
critical to the accurate portrayal of the weapon systems 
involved. The Army proponent is the most knowledgeable source 
of weapon capabilities, and should therefore work closely with 
the modellers prior to the pre-test to ensure the parameters 
are set correctly. 

Another important finding is that detections are too 
difficult to measure in the LOS-F-H field test by the use of 
video data reduction. Other battlefield occurrences, such as 
engagements and kills, can be recorded in 'real time' in a 
field test environment. Future comparisons should use a 
phenomenon other than detections that is well defined and is 
recorded by the instrumentation. 

Finally, the topic of MTM deserves further investigation. 
Future field tests of the M1A2 battle tank have already 
benefitted from the pre-test phase of MTM, and the data 
resulting from the upcoming M1A2 field tests should be 
analyzed and compared with Janus data, with the intent of 
validation or accreditation of Janus. 



38 



REFERENCES 



1. TRADOC Test and Experimentation Command, Experimentation Center, 
Line of Siaht-Forward-Heavv Initial Operational Test f LOS-F-H-IOT^ 
Test Report . Fort Ord, CA, September 1990. 

2. Bundy, D. , and Creen, M. , "Generic Model-Test-Model Using High 
Resolution Combat Models," TRADOC Analysis Command, Monterey, CA, 
unpublished . 

3. Hollis, Walter W. , "Verification and Validation (V&V) and 
Accreditation of Models", Department of the Army, Office of the 
Under Secretary, Washington, D.C., October 1989. 

4. Department of the Army, Janus fT) Documentation . TRAC-White Sands 
Missile Range, NM, June 1986. 

5. Pimper, Jeffrey E. and Dobbs, Laura A., Janus Algorithms 
Document . Lawrence Livermore National Laboratory, Livermore, CA, 
January 1988. 

6. Interview between Kelner, Al, Janus Programmer, TRAC-White Sands 
Missile Range, NM, 22 May 1991. 

7. Statistical Graphics Corporation, STATGRAPHICS . Statistical 
Graphics System . University ed. , STSC, Inc., 1988. 

8. Montgomery, Douglas C. , Design and Analysis of Experiments . John 
Wiley and Sons, 1984. 



39 



INITIAL DISTRIBUTION LIST 



No Copies 

1 . Commander 1 

U.S. Army Training and Doctrine Command 

ATTN : ATCD 

Fort Monroe, VA 23651-5000 

2. Deputy Undersecretary of the Army 1 

for Operations Research 

ATTN: Mr. Hollis 

Room 2E660, The Pentagon 

Washington, D.C. 20310-0102 

3 . Director 1 

U.S. Army TRADOC Analysis Command-Ft. Leavenworth 

ATTN: ATRC-FOQ (Technical Information Center) 

Fort Leavenworth, KS 66027-5200 

4 . Director 1 

U.S. Army TRADOC Analysis Command-WSMR 

ATTN: ATRC-WSL (Technical Library) 

White Sands Missile Range, NM 88002-5502 

5. Chief 2 

U.S. TRADOC Analysis Command-Monterey 

ATTN: CPT Dennis Bundy, USA 

P.O. Box 8692 

Monterey, CA 93943-0692 

6. Library, Code 52 2 

Naval Postgraduate School 

Monterey, CA 93943-5002 

7. Director 1 

ATTN: ATZL-TAL 

Ft. Leavenworth, KS 66027-7000 

8. Defense Technical Information Center 2 

ATTN: DTIC, TCA 

Cameron Station 
Alexandria, VA 22304-6145 



40 



9. 



1 



U.S. Army Library 

Army Study Documentation and Information Retrieval System 
ANRAL-RS 
ATTN: AS DIRS 

Room 1A518, The Pentagon 
Washington, D.C. 20310 

10. Professor Donald Barr 1 

Department of Mathematics, Code MA/Ba 

Naval Postgraduate School 
Monterey, CA 93943-5000 

11. Captain Eugene P. Paulo, USA 1 

1029 Halsey Drive 

Monterey, CA 93940 

12. Director 1 

TEXCOM Experimentation Center 

Fort Hunter Liggett, CA 93928 



41 



j Thesis 

f P272015 Paulo 
<^•1 Comparison of Janus 

and field test aircraft 
i detection ranges for 

the line-of-sight for- 
ward heavy system. 



The sir 

P272015 Paulo 

c,l Comparison of Janus 

and field test aircraft 
detection ranges for 
the line-of-sight for- 
ward heavy system. 



DUDLEY KNOX LIBRARY 




3 2768 00032018 8 











